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MULTIPLE PEN STROKE CHARACTER SET AND 
HANDWRITING RECOGNITION SYSTEM 

5 Field of the Invention 

This invention relates to computer input systems 
in general, and more specifically to an apparatus and 
handwriting alphabet for use in a handwritten input and 
recognition system used in personal computing systems 
10 such as "palm-top" computers. 

Description of Related Art 
As computers have become increasingly popular for 
various applications, portable computers have been 
developed for a wide variety of uses. While many such 

15 portable computers use a traditional keyboard for input, 
for smaller computers, particularly including hand-held 
computers, the use of "pens" as an interface has been 
introduced as a way of making a small computer easier to 
use. With a pen interface, a user can place a pen or 

20 stylus directly on a touch-sensitive screen of the 

computer to control the software running on the computer 
and to input information. For many people, controlling a 
computer and entering text with a pen is more natural 
than using a keyboard. 

25 An example of a prior art pen-based hand-held 

computer is shown in FIGURE 1. The illustrated hand-held 
computer 1 is typically about 4 inches by 6.5 inches, 
with the majority of one surface comprising a touch- 
sensitive display screen 2. The display screen 2 is 

30 typically a liquid crystal display (LCD) having a 

resolution of 256x320 pixels (although larger or smaller 
pixel arrays could be used) . Various technologies can be 
used to sense the location of a pen or stylus 3 touched 
against the surface of the LCD screen 2 to indicate to 
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the computer's operating system the X-Y coordinates of 
the touch. Various hardware buttons 4 may be provided to 
control different functions, and/ or to turn power on or 
off to the unit. In addition, a variety of software 
5 buttons or icons 5 may be provided, in known fashion, to 
indicate such functions as, for example, word processing 
or a delete function. Computer-generated information is 
typically shown on the display screen 2 as ASCII charac- 
ters 6. One such hand-held computer is available as the 

10 "Zoomer" from Casio Corporation. 

A common characteristic of such pen-based 
computers is the use of electronic "ink". "Ink" comprises 
a series or trail of pixels changed (e.g., darkened or 
lightened) as a pen 3 is moved across the display screen 

15 2 by a user, thus mimicking the application of real ink 
to paper. 

Some prior art system designers suggest the use of 
unrecognized handwritten ink input. Although this 
approach works well for recording notes for personal use, 

20 it is not always suitable for data entry into a file 

which needs to be searched at a later date. In addition, 
ink requires considerably more storage space than ASCII 
characters. Accordingly, practical pen-based computers 
need a method of inputing text which usually includes 

25 some form of recognition system. 

Various methods of recognizing handwriting are 
well known. One prior art approach is to provide a 
series of boxes in the input area (which is usually the 
display area) for entering character information. These 

3 0 systems use boxes for entry of text in an attempt to 
improve accuracy of recognition and to separate one 
character from the next. In these systems, an array of 
boxes is displayed and the user writes one character in 
each box. Although the boxes aid in improving the 

35 accuracy of recognition, most people find it awkward to 
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write in boxes. Additionally, due to the number of boxes 
necessary to capture even a short sentence, these systems 
are not very practical in a palm-top computer having a 
reduced data input area. 
5 Another character recognition system is described 

in U.S. Patent No. 5,125,039, entitled "Object 
Recognition System" , by the inventor of the present 
invention. In such a system, the user writes text 
without boxes in a free form manner. After a user inputs 

10 several ink characters, the computer applies special 
algorithms to separate the ink strokes into characters 
and then recognize each ink character as an ASCII 
character. It then replaces the ink representation of 
the characters drawn by the user with the standardized 

15 ASCII representation of those characters. Although 
these systems require less input area than boxed input 
systems, they are still difficult to implement on a 
palmtop computer having a small display. In addition, 
the computer has the additional burden of figuring out 

20 where one character ends and the next begins. This leads 
to recognition errors. 

One additional major difficulty presented by prior 
art handwriting recognition systems is the delay time 
between text input and text recognition. The prior art 

2 5 systems typically require between 2 to 5 seconds after 

the user writes the ink character on the input tablet to 
recognize and display the ASCII character on a display 
device. In typical use, the prior art systems require 
the user to write a few words and then wait several 

3 0 seconds for the computer to start the recognition 

process. Alternatively, some systems (e.g., the 
"Newton" from Apple Computer) perform recognition without 
the user stopping and waiting. But in these systems, the 
words are still recognized several seconds after they are 
35 written. In all cases, the user cannot immediately 
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realize when a recognition mistake has occurred. This 
type of handwritten text recognition system makes error 
correction difficult because the user must constantly 
look at the display for words which the user input 
5 several seconds before in order to be sure that text was 
correctly entered and correctly interpreted. Moreover, 
once a user detects an error, error correction is 
difficult because the user has to first select the word 
or characters which need to be corrected. 

10 In summary, three of the major problems with 

current handwriting recognition systems are the delay 
from writing to recognition, the limited writing area of 
palmtop computers, and the difficulty of accurately 
recognizing separate characters in non-boxed entry 

15 systems. 

Therefore, an improved pen data entry solution is 
needed which can accurately and efficiently recognize 
text on a small display. It has become evident that one 
crucial characteristic of such an improved solution is 
20 the ability to instantaneously (i.e., with little or no 
perceptible delay) recognize and display input text, 
similar to the response of currently available personal 
computers using keyboard input devices. Palm-top 
computers having the ability to instantly recognize and 

2 5 display text offer the user the opportunity to quickly 

recognize and correct mistakes. Instant recognition also 
permits the use of smaller input areas because the input 
area can be reused for writing subsequent characters. 

One of the major impediments facing "instant" 

3 0 handwritten text recognition systems is presented by the 

multiple stroke (multi-stroke) characteristic of many 
English text characters. That is, many characters 
comprise more than one pen stroke. In this context, a 
single pen stroke is defined as a continuous movement of 
35 a pen while maintaining contact with a writing tablet. 



WO 96/01453 



PCT7US95/08113 



- 5 - 

For example, the letters "T", "H," and "E" typically 
comprise multiple pen strokes, while the letters "S" and 
"O" typically comprise a single pen stroke. Prior art 
recognition systems have had difficulty in achieving 
5 essentially "instantaneous" recognition due to the fact 
that characters may comprise more than one pen stroke. 

For example, due to the possibility that any given 
input character might be a multi-stroke character, it has 
been difficult to determine when a user has completed 

10 writing a one stroke (unistroke) character, or when the 
user is continuing to write a multi-stroke character. 
For example, a vertical line might represent the letter 
"I" or it could represent the first stroke in the multi- 
stroke letters "T", "H" or "E" . In the past, recognition 

15 systems have solved this ambiguity by waiting until the 
user stopped writing, or by having a fixed delay period 
after which characters were recognized, or by detecting 
the start of a next stroke sufficiently far from prior 
strokes as to indicate a new character. Each of these 

20 approaches are deficient due to the recognition time 
delays introduced. 

Recently, two approaches have been attempted for 
immediate recognition of handwritten text. Neither of 
these two approaches has proven wholly satisfactory. The 

25 first approach is offered by Sharp Electronics of Japan 
in their PVF1 handheld computer system, which provides 
"immediate" recognition of both English and Japanese 
characters. The Sharp system uses a modified boxed input 
method. It displays several adjacent boxes on a screen 

3 0 for text input. Every text character is written into one 
of the boxes. Recognition timing delays are reduced 
because the system knows to begin recognizing a character 
previously written into a first entry box as soon as the 
user begins writing into an another entry box. The 

35 recognized character is subsequently displayed upon the 
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screen (not in the box) as soon as the recognition 
process completes. Expert users can quickly enter 
multiple characters by alternating between two adjacent 
boxes. This is different from previous boxed input 
5 systems where the user wrote characters in a left to 
right fashion in a multitude of boxes. The Sharp 
approach achieves faster recognition response on a 
reduced display area than previous systems. However, it 
suffers from several disadvantages, 

10 Although the Sharp system uses fewer boxes (as 

little as two will suffice) , the boxes still occupy a 
significant amount of screen area. In addition, as with 
all boxed input systems, the user has to be careful to 
always write within the box. If one stroke of a multi- 

15 stroke character falls outside the box, the character 
will be recognized incorrectly. This requires the user 
to carefully look at the screen at all times while 
writing. Another, and more serious drawback, is that the 
recognition of characters is not completely "instant". 

20 In this system, recognition of one character does not 
commence until the user starts writing a subsequent 
character. Although this system represents an 
improvement over the prior art systems where recognition 
delays were longer, recognition is still delayed. So, 

25 when the user writes just one character, or when the user 
writes the last character in a sequence, that character 
is not recognized until after a pre-determined time-out 
delay. This delay after writing a single character makes 
it frustrating and therefore impractical to make quick 

3 0 editing changes such as writing a "backspace" character, 
or to insert a single character. 

A second approach at immediate recognition of 
handwritten text was recently described by Xerox 
Corporation of Palo Alto, CA. Xerox teaches a method 

35 whereby every character that a user wishes to write is 
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represented by a single stroke glyph. Because every 
character is represented using a single stroke, 
recognition commences as soon as the user lifts the pen 
from the writing tablet. The system provides improved 
5 recognition speeds over the Sharp approach and avoids the 
problems associated with the writing boxes used in the 
Sharp system. However, the Xerox method suffers from 

two major disadvantages. First, the Xerox approach is 
difficult to learn because it requires the user to 

10 memorize an entirely new alphabet for entering text. The 
alphabet is specially designed to maximize the 
recognition abilities of the computer, not to maximize 
ease of learning. The Xerox disclosure recognizes this 
difficulty yet submits that the inefficiency of learning 

15 the alphabet is compensated by the improved recognition 
speeds once the user becomes an expert. 

Second, the Xerox approach is difficult to 
implement with a full set of characters. The user must 
learn a single stroke representation for every possible 

2 0 character. Although this task may be feasible when 

representing only the 2 6 letters of the English alphabet 
in one case (upper or lower) , there are many more 
characters requiring representation and recognition. For 
example, both upper and lower case English characters 
25 must be recognized. European languages have multiple 
accented characters as well as many other unique 
characters. In addition, a myriad of punctuation marks 
and mathematical symbols require representation. 
Assigning each of these characters to a unique single 

3 0 stroke glyph requires inventing many strange and novel 

glyphs that are non-intuitive and therefore difficult to 
learn by the average user. Compounding this difficulty 
is the problem of similarly looking accented characters 
(for example, A, A, A, A, and A) . Assigning unique 
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glyphs for these characters would make the extended 
alphabet especially non-intuitive and difficult to learn. 

The limitations of a unistroke alphabet as taught 
by Xerox are magnified when trying to create an immediate 
5 recognition system for Asian languages. For example, it 
is nearly impossible to define single stroke alphabets 
for Asian symbols, such as Japanese katakana or hiragana, 
Chinese kanji, or Korean hangul, due to the large number 
of symbols that need to be represented. 

10 Accordingly, there is a need for an improved 

handwritten text recognition system capable of 
instantaneously and accurately recognizing handwritten 
text entries. There is also a need for an improved 
handwritten text entry and recognition system which is 

15 user-friendly, easy to learn, and easy to implement. 

The present invention provides such a handwritten 
text recognition system. 

SUMMARY OF THE INVENTION 

The present invention uses a pen or stylus as an 
2 0 input device to a pen-based computer handwriting 

recognition system capable of interpreting a special pre- 
defined set of character strokes or glyphs. The 
invention teaches a system which provides true immediate 
character recognition, yet allows characters to be 
25 written with any number of strokes, thus making it 

natural to use and easy to learn. The present invention 
defines three different categories of pen strokes: (1) 
pre-character modifier strokes, (2) character or symbol 
strokes, and (3) post-character modifier strokes. 
30 Pre-character modifier strokes precede character 

strokes and inform the present recognition system that 
subsequently entered character strokes are to be modified 
by the pre-character modifier stroke in a defined manner. 
They function primarily to control the interpretation of 
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a subsequently entered character stroke. For example, a 
pre-modifier control stroke may indicate that the next 
character stroke is to be interpreted as a punctuation 
character. Pre-character modifier strokes may or may not 
5 cause an immediate visible display change. In the 
preferred embodiment of the invention, pre-character 
modifier strokes do result in a display change (by either 
changing a status indicator or by displaying a temporary 
character) , so the user knows the pre-character modifier 

10 stroke was successfully entered. 

Character strokes always cause a letter or other 
symbol to be displayed the moment the stroke is input on 
the writing tablet, interpreted in accordance with any 
pre-character modifier strokes previously entered. Any 

15 status indicators or temporary characters displayed due 
to earlier pre-character modifier strokes are removed 
upon recognizing a character stroke. 

Post-character modifier strokes cause the 
recognition system to modify, in a defined manner, a 

2 0 character or symbol which was previously entered and 

displayed. For example, a post-character modifier may be 
used to add a diacritical mark to a character. 

An important advantage of the present invention is 
its ability to recognize characters consisting of 

2 5 multiple pen strokes yet still provide instantaneous 

recognition and display of the recognized character. By 
combining mutually exclusive pre-character modifier 
strokes, character strokes, and post-character modifier 
strokes, a myriad of alpha, numeric, punctuation, and 

3 0 accented characters may be entered with natural and easy 

to learn styles. 

The use of the three different types of strokes 
guarantees that the system always knows whether the user 
is starting a new character, has completed a character, 
3 5 or is modifying a previously recognized character. This 
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enables the system to provide the immediate response that 
is desired. 

It will be shown that the present invention is 
flexible and can be used to enter not only English and 
5 other Roman character-based languages, but other written 
alphabets, such as Japanese hiragana and katakana. 

The details of the preferred embodiment of the 
present invention are set forth in the accompanying 
drawings and the description below. Once the details of 
10 the invention are known, numerous additional innovations 
and changes will become obvious to one skilled in the 
art. 



BRIEF DESCRIPTION OF THE DRAWINGS 

FIGURE 1 is a front left-side perspective drawing 
15 showing a prior art pen-based hand-held computer. 

FIGURE 2 is a flow chart describing the preferred 
embodiment of the handwriting recognition system of the 
present invention . 

FIGURE 3 shows the pen strokes used to represent 

2 0 the 2 6 letters of the ordinary English alphabet in a 

prior art system taught by Xerox. 

FIGURE 4a shows the pen strokes used to represent 
the 2 6 letters of the ordinary English alphabet in the 
preferred embodiment of the present invention. 
25 FIGURE 4b shows the pen strokes used to represent 

the 10 numbers of the Arabic number system in the 
preferred embodiment of the present invention. 

FIGURE 5a shows the pen strokes used as character 
strokes to represent three common "non-printing" 

3 0 characters (the space, the backspace, and the carriage 

return) in the preferred embodiment of the present 
invention. 

FIGURE 5b shows the pen strokes used as pre- 
character modifier strokes to create capitalized 
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characters, punctuation, and extended characters in the 
preferred embodiment of the present invention. 

FIGURE 5c shows the pen strokes used to represent 
common punctuation symbols when preceded by the pre- 
5 character modifier stroke for punctuation in the 
preferred embodiment of the present invention. 

FIGURE 6 shows the pen strokes used to represent 
several extended symbols when preceded by the pre- 
character modifier stroke for extended symbols in the 
10 preferred embodiment of the present invention. 

FIGURE 7a shows the pen strokes used in the 
preferred embodiment of the present invention as post- 
character modifier strokes to add accents to letters 
created with a character stroke and none or more pre- 
15 character modifier strokes. 

FIGURE 7b shows several examples of writing 
accented characters using multiple pen strokes used in 
the preferred embodiment of the present invention. 

FIGURE 7c shows an example of writing an upper 

2 0 case accented character using a pre-character modifier 

stroke, a character stroke, and a post-character modifier 
stroke. 

FIGURE 8 shows the dictionary mapping used to 
enter katakana characters. Each of these entries 
25 consists of either a single character stroke, a pre- 
character modifier stroke combined with a character 
stroke, or two pre-character modifier strokes combined 
with a character stroke. This mapping follows the well 
known romaji input system. 

3 0 FIGURE 9 shows the sequence of strokes and the 

resulting display when entering a three stroke katakana 
character. 

FIGURE 10 shows the dictionary mapping used to 
enter special two katakana character sequences. Each of 
35 these entries consists of two pre-character modifier 
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strokes and one character stroke. This mapping follows 
the well known romaji input system. 

FIGURE 11a shows named pen strokes useful for 
defining various symbols in the preferred embodiment of 
5 the present invention. 

FIGURE lib shows variations of the pen strokes 
that can be used to represent the 2 6 letters of the 
ordinary English alphabet in the preferred embodiment of 
the present invention. 
10 FIGURE 11c shows the pen strokes used as character 

strokes to represent common "non-printing" characters in 
the preferred embodiment of the present invention. 

FIGURE lid shows the pen strokes used as character 
strokes to represent common punctuation characters in the 
15 preferred embodiment of the present invention. 

FIGURE lie shows the pen strokes used as character 
strokes to represent additional punctuation characters in 
the preferred embodiment of the present invention. 

FIGURE llf shows the pen strokes used as character 
2 0 strokes to represent extended characters in the preferred 
embodiment of the present invention. 

FIGURE llg shows the pen strokes used as character 
strokes to represent non-accented foreign characters in 
the preferred embodiment of the present invention. 
25 Like reference numbers and designations in the 

various drawings refer to like elements. 



DETAILED DESCRIPTION OF THE INVENTION 

Throughout this description, the preferred 
embodiment and examples shown should be considered as 
30 exemplars, rather than as limitations on the present 
invention. 
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Overview 

The present invention is preferably implemented as 
a computer program operating on a pen-based computer 
system such as the "Zoomer" and "Newton" products 
5 described above. The computer program is stored on a 
storage media or device readable by a computer, and 
configures and operates the computer when the storage 
media or device is read by the computer, the computer 
being operated to determine, recognize, classify, and 
10 sometimes display handwritten strokes. However, as is 
known in the art, the logic functions of such a computer 
program may be implemented as an electronic system, such 
as a firmware programmed, microcoded, or hardwired logic 
system. 

15 FIGURE 2 is a flow chart that describes the basic 

process of the invention. Operation starts at step 100, 
where a pen stroke is received. A stroke is a movement 
by a user of a pen or equivalent input device on a 
tablet, pad, or digitizer. A stroke begins when the pen 

2 0 touches the tablet, and ends when the user removes the 
pen from the tablet. The position and movement of the 
pen during the stroke is converted into a series of X-Y 
coordinates. This pen-position data is received from the 
pen input device and passed to the glyph recognizer 

25 portion of the character input system. 

In step 101, the glyph recognizer logic recognizes 
the pen-stroke data as being a rendition of a particular 
glyph based on the movement and shape of the input 
stroke. The recognition procedure used by the present 

30 invention is essentially that disclosed in U.S. Patent 

No. 5,125,039, entitled Object Recognition System, issued 
to Jeffrey Hawkins, one of the present inventors. This 
patent is incorporated herein by reference. 

In step 102, a dictionary is consulted to look up 

35 how the current glyph is to be interpreted. Associated 
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with each glyph is a definition, which includes the 
current classification of the glyph as a pre-character 
modifier stroke, a character (meaning any symbol) stroke, 
a post-character modifier stroke, or as a currently 
5 unassigned stroke. The definition of each glyph also 

includes what character or other indicator, if any, is to 
be output in response to a stroke that matches that 
glyph. It also includes a specification of what changes 
if any, are to be made to any glyph definitions in 

10 response to a stroke that matches that glyph. 

Steps 103 and 104 are decision steps. Different 
actions are taken depending on whether the current stroke 
represents a pre-character modifier stroke, a character 
stroke, a post-character modifier stroke, or an 

15 unassigned stroke. In the case of pre-character modifier 
strokes, control passes to step 2 00. 

In step 200, the processing logic causes an 
indication to be made that a particular pre-character 
modifier stroke has been input. While the pre-character 

2 0 modifier stroke does not result in a character being 

output from the character recognizer, it is nevertheless 
desirable to display to the user an indicator of what 
pre-character modifier stroke was received. In the 
preferred embodiment of the invention a character is 

25 displayed which is representative of the pre-character 

modifier stroke. For example, if a simple "tap" or "dot" 
pre-character modifier stroke is used to interpret the 
next character stroke as a punctuation symbol, then a 
bullet or "dot 11 character could be displayed indicating 

30 that the dot stroke was entered successfully. These 

characters are temporary and will be removed in step 300. 

In step 201, the definition in the glyph 
dictionary of one or more glyphs is modified according to 
the input pre-character modifier stroke, as further 

35 described below. 
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In step 3 00, if the class of the current stroke is 
"character", then any previously displayed indicators or 
characters representing pre-character modifier strokes 
are deleted and removed from the display. 
5 The definition of the corresponding glyph in the 

dictionary includes the character that is to be output in 
response to receiving strokes matching that glyph. In 
step 3 01, the processing logic outputs that character for 
display. 

10 In step 302, the definition in the glyph 

dictionary of one or more glyphs is modified according to 
the character stroke entered and the current state of the 
dictionary, as further described below. 

In step 400, if the class of the current pen 

15 stroke is "post-character modifier", then the processing 
logic causes the most recently output character to be 
removed and replaced with a new or modified character. 
In the preferred embodiment, the prior character is 
removed by sending a "backspace" character to the display 

2 0 system. The new or modified character is determined by 

the glyph definition corresponding to the post-character 
modifier stroke. 

In step 401, the definition in the glyph 
dictionary of one or more glyphs is modified according to 
25 the post-character modifier stroke entered and the 

current state of the dictionary, as further described 
below. 

At any point in time, the glyph dictionary 
specifies exactly one interpretation for each input glyph 
30 that the system knows how to recognize. It is possible 
for that interpretation to be to ignore that glyph. For 
example, the glyph which looks like a pair of eyeglasses 
(see FIGURE 5c) is not active in the initial dictionary 
definitions. When a stroke corresponding to that glyph is 

3 5 received when that glyph is not active, the stroke is 
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recognized, but ignored. At other times however, the 
eyeglasses glyph will not be ignored. For example, after 
the recognition of a character stroke for a "u" or "a", 
the eyeglasses glyph becomes active as a post-character 
5 modifier stroke and corresponds to the umlaut accent. If 
the eyeglasses glyph is then input, the original "u" or 
"a" input character will be changed to "u" or "a". 
Similarly, after a pre-character modifier stroke for 
punctuation, the eyeglasses glyph is defined as a 

10 character stroke representing the percent "%" character. 
At any point in time, each stroke that is 
recognized must be a pre-character modifier stroke, a 
character stroke, a post-character modifier stroke, or 
unassigned. It cannot at a any point in time have 

15 multiple classifications. Nevertheless, any stroke, when 
recognized, may modify the definition of any or all 
glyphs so as to reclassify them. Thus, a particular 
glyph may correspond sometimes to a character stroke, and 
at other times to a post-character or pre-character 

2 0 modifier stroke. 

Steps 201, 302, and 401 all modify the dictionary 
to reassign the definitions of glyphs. There are numerous 
ways the dictionary can be organized to allow for this 
modification. In the preferred embodiment of the present 
25 invention, the dictionary is implemented as a tree data 
structure. The root level of the tree contains 
definitions for all the pre-character modifier strokes 
and character strokes defined in the initial state of the 
system. When one of these strokes is entered, there may 

3 0 be a subsequent branch defined for that stroke which 

contains new definitions for all the glyphs. Each stroke 
leads to a new point in the tree data structure, thus 
modifying the dictionary. Implementing the dictionary as 
a tree data structure is flexible and allows the 
3 5 dictionary to be located in ROM. Other methods for 
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organizing the dictionary would be possible, as known in 
the art. 

FIGURE 3 shows the glyphs taught in the prior art 
referenced earlier from Xerox for entering the English 
5 alphabet (in this and subsequent figures, a dot at one 
end of a stroke indicates that the stroke is drawn 
starting at that end,) This system achieves immediate 
recognition by forcing every character to be input with 
only a single stroke. This rigid unistroke approach 

10 makes it difficult to extend the character set 

significantly beyond the base alphabet. Xerox does not 
teach any method for punctuation, accented characters, 
extended characters, upper case letters, numbers, or 
other non-Roman alphabets. The Xerox alphabet was also 

15 designed for speed of entry and simplicity of 

recognition. This results in a glyph set that looks 
unfamiliar and is difficult to learn. 

FIGURE 11a shows named pen strokes useful for 
defining various symbols in the preferred embodiment of 

2 0 the present invention. These strokes, variations of 

these strokes, and other strokes, as character strokes, 
pre-character modifier strokes, and post-character 
modifier strokes, can be defined to represent essentially 
any character or symbol. 

25 FIGURE 4a shows the glyph set used in the 

preferred embodiment of the present invention for 
entering the English alphabet. This alphabet was chosen 
to be as familiar as possible and easy to learn, yet 
adhere to the principles taught in this invention. Most 

30 of these alphabet characters are written with a single 

character stroke. However, the present invention teaches 
a method of achieving immediate recognition with multiple 
strokes. It was found through user testing that most 
people had difficulty learning any single stroke "X". 

3 5 Therefore, in the base alphabet, "X" is written with two 
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sequential strokes. The first stroke going from the 
upper left to the lower right is a pre-character modifier 
stroke, and the second stroke going from the upper right 
to the lower left is a character stroke. User testing 
5 has shown that this two stroke combination is far easier 
to write than any one stroke "X" . 

The alphabet shown in FIGURE 4a provides near 100% 
recognition accuracy, yet is easy to learn and use due to 
its obvious similarity to natural handwriting styles, 

10 FIGURE lib shows that there are actually multiple 

ways (different strokes) which can be used to write many 
of these letters, making the system even easier to learn 
and use. The recognition system simply maps the input 
strokes for these letters to the same output symbol in 

15 the glyph dictionary. For example, there are two ways to 
write a "Y" : the glyph shown in FIGURE 4a, and a shape 
similar to a lower case scripted "Y" as shown in FIGURE 
lib. User testing has shown that some users prefer one 
method and some prefer the other. 

2 0 FIGURE 4b shows the glyph set used in the 

preferred embodiment of the present invention for 
entering the digits 0 through 9. Many of these glyphs 
are also used to enter letters of the alphabet. One 
method of overcoming this ambiguity problem is to have a 
25 separate numeric mode where a user only can enter digits. 
This numeric mode can be entered by pressing a button on 
the display of the computer, by writing a "num-lock" 
glyph, or other means. In the preferred embodiment of 
the present invention, a user can enter and exit a 

3 0 numeric mode by either tapping an on-screen icon or by 

writing a "forward slash" stroke (a slanted line written 
from bottom left to upper right) . Testing has shown that 
occasionally a user forgets to exit numeric mode after 
writing several digits. A refinement of the present 
35 invention helps fix this problem by automatically exiting 
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numeric mode when the user writes a character stroke 
which can only be interpreted as a letter or when the 
user writes the pre-character modifier stroke for capital 
letter shift. 

5 FIGURE 5a shows the glyph set used in the 

preferred embodiment of the present invention for 
entering three common "non-printing" characters: the 
space, the backspace, and the carriage return. A 
recognition system with immediate response operates in 

10 many ways more like a keyboard than a conventional 

handwriting recognition system. For example, because 
users see results instantly, they find it more natural to 
"backspace" over an incorrect character than write over 
it or delete it with a gesture. Therefore, simple 

15 character strokes may be provided for space, backspace, 
carriage return, and other keyboard equivalents. FIGURE 
11c details other non-printing keyboard equivalents. 

FIGURE 5b shows the glyph set used in the 
preferred embodiment of the present invention as pre- 

2 0 character modifier strokes for the English language. The 

three strokes are used to respectively indicate that the 
subsequent character stroke represents a capital letter, 
a punctuation character, or an extended character. If 
desired, other pre-character modifier strokes may be 
25 defined as "sticky" shift strokes that take effect until 
reset. For example, a "caps lock" stroke would cause all 
subsequent strokes to be interpreted as if the pre- 
character modifier "caps" stroke had been entered before 
each subsequent stroke. 

3 0 FIGURE 5c shows the glyph set used in the 

preferred embodiment of the present invention for 
entering common punctuation symbols. All of these glyphs 
are classified as character strokes when preceded by a 
pre-character modifier stroke representing punctuation 
35 characters. As shown in FIGURE 5b, a "dot" stroke is 
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used in the preferred embodiment of the invention to 
indicate a punctuation character. A more complete list 
of preferred punctuation characters is shown in FIGURE 
lid and FIGURE lie* 
5 FIGURE 6 shows the glyph set used in the preferred 

embodiment of the present invention for entering several 
extended symbols. All of these glyphs are classified as 
character strokes when preceded by a pre-character 
modifier stroke representing extended symbol characters. 

10 As shown in FIGURE 5b, a "slash" stroke is used in the 
preferred embodiment of the invention to indicate an 
extended symbol character. A more complete list of 
preferred extended symbols is shown in FIGURE llf. 

FIGURE 7a shows the glyph set used in the 

15 preferred embodiment of the present invention for adding 
accents or diacritical marks to letters. These glyphs 
are classified as post-character modifier strokes after 
entering a letter which is capable of being accented. 

FIGURE 7b shows several examples of how a user 

2 0 would write an accented character using a two stroke 
combination of a character stroke and a post-character 
modifier stroke. Writing these accented characters using 
the present invention is very similar to how the user 
would write them on paper. First, the user writes a base 

25 letter, then adds an accent. After writing the base 

letter stroke, the corresponding character is immediately 
output to the display. Upon writing the accent post- 
character modifier stroke the base letter is erased and 
replaced with the correct accented letter. Thus, the 

30 system achieves an immediate recognition response on a 
multiple stroke character even though it is initially 
unknown whether the user is writing a single stroke 
letter or a multiple stroke accented letter. 

FIGURE 7c shows an example of how a user would 

35 write an accented upper case character using a three 



WO 96/01453 



PCT/US95/08113 



stroke combination of a pre-character modifier stroke, a 
character stroke, and a post-character modifier stroke. 
The strokes are numbered in the order they are written. 
First, the user writes a pre-character modifier stroke 
5 indicating that the following character should be 
capitalized. This causes the display of a temporary 
character indicating the acceptance of the stroke, in 
this case an "up arrow" character. Next, the user writes 
a character stroke. The temporary character is removed 

10 and replaced by the appropriate upper case letter. 

Lastly, the user writes a post-character modifier stroke, 
causing the base upper case letter to be erased and 
replaced with the correct upper case accented letter. 

The present invention is quite flexible and can 

15 accommodate many different types of languages, alphabets, 
and symbol systems. FIGURE 8 illustrates one method of 
how the invention can be adapted to quickly and easily 
enter the Japanese katakana alphabet. In Japan, a common 
method for entering katakana with a keyboard is called 

20 the -"romaji" method. In the romaji method, a user types 
the English phonetic equivalent of the katakana 
characters. For example, to enter the katakana for 
"sushi", the user types "su" which results in the display 
of the corresponding katakana symbol. Then the user 

25 types "shi", which results in the display of the 

corresponding katakana symbol. All of the approximately 
90 katakana characters can be input with combinations of 
one, two, or three typed letters. The present invention 
can duplicate this method simply with a change to the 

3 0 dictionary of glyph mappings. In the preferred embodiment 
of a katakana stroke recognition system in accordance 
with the present invention, the new dictionary assigns 
strokes representing the consonants "BCDGHKMNPRSTXYZ" as 
pre-character modifier strokes and assigns strokes 

35 representing the vowels "AEIOU" as character strokes. 
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Some katakana characters are entered with just one 
character stroke. Some katakana characters are entered 
with one pre-character modifier stroke and one character 
stroke. Some katakana characters are entered with two 
5 pre-character modifier strokes and one character stroke. 
In the latter case, two temporary indicator characters 
are preferably displayed, representing the two pre- 
character modifier stokes. Both temporary characters are 
deleted and replaced with the final katakana character 

10 upon entering the character stroke. This sequence is 
illustrated in FIGURE 9. 

FIGURE 10 shows how special double katakana symbol 
combinations can be entered with three stroke 
combinations of two pre-character modifier strokes and 

15 one character stroke. This mapping still follows the 
romaji method common in Japan. It illustrates the 
flexibility of the present invention by showing how a 
character stroke can result in the display of more than 
one character or symbol. In principle, a character stroke 

2 0 or post-character modifier stroke can result in the 

output and display of any length sequence of characters. 

There are many fine points in this particular 
implementation which are not detailed here but would be 
obvious to anyone experienced in the romaji input system 
25 and familiar with the present invention. For example, 
the stroke representing the letter "N" is initially a 
pre-character modifier stroke. Once entered it is 
reassigned as a character stroke. This permits the entry 
of the "NN" katakana symbol, which is an exception to the 

3 0 general consonant-vowel pairing for katakana. 

FIGURE llg shows the pen strokes used as character 
strokes to represent non-accented foreign characters in 
the preferred embodiment of the present invention. 

The principles of the present invention can be 
35 extended to other characters sets, such as Japanese 
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hiragana, Chinese kanji, or Korean hangul. The concepts 
of the present invention also can be extended to provide 
greater flexibility for users. For example, the computer 
system could be programmed to allow a user to define new 
5 input strokes , and/or to associate symbols, characters, 
or even complete words or phrases, to a combination of 
input strokes. Thus, a user-maintained glossary could be 
built where the user could define the sequences of 
characters — or symbols, text, or program functions — 

10 to be associated with a stroke, a multi-stroke 

combination, or sequence of multiple stroke combinations. 
Alternatively, the user could also define new strokes 
within a table (or other data structure) and assign 
context to each such stroke. 

15 Summary 

The present recognition system provides several 
improvements over prior art systems by achieving 
immediate recognition of multiple stroke characters 
without using boxes for input. It improves over prior 

2 0 art systems by immediately displaying every character 
written as soon as the user finishes the character. No 
unnecessary delays are incurred, nor are additional 
actions required of a user to translate input. The 
immediate response helps a user to quickly identify 

25 mistakes and correct them. The present system 

accomplishes this immediate response while at the same 
time accommodating characters which are more easily 
learned and written using multiple strokes. Examples 
provided above include accented characters, punctuation 

30 characters, extended symbols, the letter "X", and the 
katakana character set. Defining characters with 
multiple strokes makes the present invention much easier 
to use and learn than prior art systems requiring single 
stroke only characters. 
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Another advantage provided by the present 
recognition system is that large sets of characters can 
be represented without relying on a large set of unique 
strokes or glyphs. For example, accented letters use the 
5 same base character stroke as their unaccented 

counterpart letters. Similarly, punctuation marks and 
capitalized letters are realized using a combination of 
pre-modifier control strokes and character strokes. The 
present invention provides an additional advantage over 
10 prior art systems because the present system does not 
require multiple writing boxes or other large on-screen 
gadgetry. Valuable display space is thereby saved, 
allowing the present system to be used on very small 
devices . 

15 The present system also teaches an alphabet for 

inputing Roman-character based languages which, although 
not identical to a user's conventional writing style, is 
very similar and therefore easy to learn. A user can 
achieve near 100% recognition accuracy when using this 

2 0 alphabet, yet it is very easy to learn and use because it 
is very similar to a natural writing style. 

A number of embodiments of the present invention 
have been described. Nevertheless, it will be understood 
that various modifications may be made without departing 

2 5 from the spirit and scope of the invention. For example, 

while particular strokes and associations for such 
strokes have been disclosed, the invention encompasses 
other strokes, combinations, and associations. Further, 
as noted above, a stroke may be associated with any 

3 0 context, such as character (s) , symbol (s), text, or 

program functions. The term "character" should thus be 
understood to encompass any of these contexts. 
Accordingly, it is to be understood that the invention is 
not to be limited by the specific illustrated embodiment, 
3 5 but only by the scope of the appended claims. 
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1. An improved electronic handwriting 
recognition system for interpreting input strokes and 
displaying corresponding characters, the system being of 
the type having an input writing surface, a device for 

5 inputing strokes on the input writing surface, and a 
display for displaying characters, the improvement 
comprising: 

(a) recognition logic for recognizing and 

classifying each individual input stroke 
10 after input as a member of one of a plurality 

of sets of glyphs, the sets comprising at 
least: 

(1) a set of character glyphs; 

(2) a set of post-character modifier glyphs; 
15 (b) processing logic, coupled to the recognition 

logic, for output ing to the display a 
character corresponding to a character glyph, 
wherein the processing logic modifies the 
displayed character in a pre-defined manner 
20 in response to any immediately succeeding 

post-character modifier glyphs. 

2. An improved electronic handwriting 
recognition system for interpreting input strokes and 
displaying corresponding characters, the system being of 

25 the type having an input writing surface, a device for 
inputing strokes on the input writing surface, and a 
display for displaying characters, the improvement 
comprising: 

(a) recognition logic for recognizing and 
30 classifying each individual input stroke 

after input as a member of one of a plurality 
of sets of glyphs, the sets comprising at 
least: 

(1) a set of pre-character modifier glyphs; 
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(2) a set of character glyphs; 

(3) a set of post-character modifier glyphs; 
(b) processing logic, coupled to the recognition 

logic, for outputing to the display a 
5 character corresponding to a character glyph, 

wherein the processing logic modifies the 
character to be displayed in a pre-defined 
manner in response to any immediately 
preceding pre-character modifier glyphs, and 
10 wherein the processing logic modifies the 

displayed character in a pre-defined manner 
in response to any immediately succeeding 
post-character modifier glyphs. 



3. An automated method for interpreting input 
15 strokes and displaying corresponding characters on an 
electronic handwriting recognition system, the system 
comprising an input writing surface, a device for 
inputing strokes on the input writing surface, and a 
display for displaying characters, the method comprising 
20 the steps of: 

(a) recognizing and classifying each individual 
input stroke after input as a member of one 
of a plurality of sets of glyphs, the sets 
comprising at least: 

25 (1) a set of character glyphs; 

(2) a set of post-character modifier glyphs; 

(b) outputing to the display a character 
corresponding to a character glyph, wherein 
the processing logic modifies the displayed 

3 0 character in a pre-defined manner in response 

to any immediately succeeding post-character 
modifier glyphs. 
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4 . An automated method for interpreting input 
strokes and displaying corresponding characters on an 
electronic handwriting recognition system, the system 
comprising an input writing surface, a device for 
5 inputing strokes on the input writing surface, and a 

display for displaying characters, the method comprising 
the steps of: 

(1) recognizing and classifying each individual 
input stroke after input as a member of one 
10 of a plurality of sets of glyphs, the sets 

comprising at least: 

(1) a set of pre-character modifier glyphs; 

(2) a set of character glyphs; 

(3) a set of post-character modifier glyphs; 
15 (b) outputing to the display a character 

corresponding to a character glyph, wherein 
the processing logic modifies the character 
to be displayed in a pre-defined manner in 
response to any immediately preceding pre- 
20 character modifier glyphs, and wherein the 

processing logic modifies the displayed 
character in a pre-defined manner in response 
to any immediately succeeding post-character 
modifier glyphs. 

25 5. A storage media readable by a programmable 

computer when coupled to the storage media, the storage 
media containing a control program tangibly stored 
thereon, such that the computer is operated by the 
control program when the storage media is read by the 

3 0 computer, the computer being operated to determine, 

recognize, and classify handwritten strokes, the control 
program being configured to operate the computer to 
perform the functions of: 
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(a) recognizing and classifying each individual 
input stroke after input as a member of one 
of a plurality of sets of glyphs, the sets 
comprising a set of character glyphs and at 

5 least one of: 

(1) a set of pre-character modifier glyphs; 

(2) a set of post-character modifier glyphs ; 

(b) outputing to the display a character 
corresponding to a character glyph, wherein 

10 the processing logic modifies the character 

to be displayed in a pre-defined manner in 
response to any immediately preceding pre- 
character modifier glyphs, and wherein the 
processing logic modifies the displayed 

15 character in a pre-defined manner in response 

to any immediately succeeding post-character 
modifier glyphs. 



6. A control program tangibly stored on a 
storage media readable by a programmable computer, such 

2 0 that the computer is operated by the control program when 
the storage media is read by the computer, the computer 
being operated to determine, recognize, and classify 
handwritten strokes, such functions being performed by 
the combination of the control program and the computer 

25 performing the functions of: 

(a) recognizing and classifying each individual 
input stroke after input as a member of one 
of a plurality of sets of glyphs, the sets 
comprising a set of character glyphs and at 

30 least one of: 

(1) a set of pre-character modifier glyphs; 

(2) a set of post-character modifier glyphs; 

(b) outputing to the display a character 
corresponding to a character glyph, wherein 
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the processing logic modifies the character 
to be displayed in a pre-defined manner in 
response to any immediately preceding pre- 
character modifier glyphs, and wherein the 
5 processing logic modifies the displayed 

character in a pre-defined manner in response 
to any immediately succeeding post-character 
modifier glyphs. 



7. The invention of claims 1 or 2 , wherein the 
10 recognition logic and processing logic are embodied in a 

pen-based portable computer. 

8. The invention of claim 2, wherein the 
processing logic outputs an indicator to the display in 
response to recognition and classification of a pre- 

15 character modifier glyph, and removes the indicator in 

response to recognition and classification of a character 
glyph. 

9. The invention of claims 1, 2, 3, 4, 5, or 6, 
wherein modification of the displayed character is 

20 accomplished by outputing a "backspace" character to 
delete the displayed character and then outputing a 
modified character to replace the deleted character. 

10. The invention of claims 1, 2, 3, 4, 5, or 6, 
wherein recognition of a stroke modifies the 

25 classification of at least one subsequent stroke. 

11. The invention of claims 2, 4, 5, or 6, 
wherein recognition of a pre-character modifier stroke 
modifies the definition of the character corresponding to 
at least one subsequently input character glyph. 
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12. The invention of claim 4, further including 
the steps of: 

(a) outputing an indicator to the display in 
response to recognition and classification of 

5 a pre-character modifier glyph; and 

(b) removing the indicator in response to 
recognition and classification of a character 
glyph. 



13. The invention of claims 5 or 6, further 
10 including the functions of: 

(a) outputing an indicator to the display in 
response to recognition and classification of 
a pre-character modifier glyph; and 

(b) removing the indicator in response to 

15 recognition and classification of a character 

glyph. 

14. A set of glyphs for use in conjunction with 
an electronic handwriting recognition system, the system 
comprising an input writing surface, a device for 

2 0 input ing strokes on the input writing surface, and a 

display for displaying characters, wherein each stroke 
glyph corresponds to a unique alphabetic character, the 
set of glyphs and corresponding characters being 
substantially as shown in FIGURE 4a. 

25 15. A set of glyphs for use in conjunction with 

an electronic handwriting recognition system, the system 
comprising an input writing surface, a device for 
inputing strokes on the input writing surface, and a 
display for displaying characters, wherein each stroke 

3 0 glyph corresponds to a unique alphabetic vowel character, 

the set of glyphs and corresponding vowel characters 
being substantially as shown in FIGURE 4a. 
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16. A set of glyphs for use in conjunction with 
an electronic handwriting recognition system, the system 
comprising an input writing surface, a device for 
inputing strokes on the input writing surface, and a 

5 display for displaying characters, wherein each stroke 
glyph corresponds to a unique numeric character, the set 
of glyphs and corresponding characters being 
substantially as shown in FIGURE 4b. 

17. A set of glyphs for use in conjunction with 
10 an electronic handwriting recognition system, the system 

comprising an input writing surface, a device for 
inputing strokes on the input writing surface, and a 
display for displaying characters, wherein each stroke 
glyph corresponds to a unique non-printing character, the 
15 set of glyphs and corresponding characters being 
substantially as shown in FIGURE 5a. 

18. A set of glyphs for use in conjunction with 
an electronic handwriting recognition system, the system 
comprising an input writing surface, a device for 

2 0 inputing strokes on the input writing surface, and a 

display for displaying characters, wherein each stroke 
glyph corresponds to a unique pre-character modifier 
stroke, the set of glyphs and corresponding modifier 
strokes being substantially as shown in FIGURE 5b. 

25 19. A set of glyphs for use in conjunction with 

an electronic handwriting recognition system, the system 
comprising an input writing surface, a device for 
inputing strokes on the input writing surface, and a 
display for displaying characters, wherein each stroke 

3 0 glyph corresponds to a unique post-character modifier 

stroke, the set of glyphs and corresponding modifier 
strokes being substantially as shown in FIGURE 7a. 
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20. An improved electronic handwriting 
recognition system for interpreting input strokes and 
displaying corresponding non-Roman characters, the system 
being of the type having an input writing surface, a 
5 device for inputing strokes on the input writing surface, 
and a display for displaying characters, the improvement 
comprising: 

(a) recognition logic for recognizing and 

classifying each individual input stroke 
10 after input as a member of one of a plurality 

of sets of glyphs, the sets comprising at 
least: 

(1) a set of pre-character modifier glyphs; 

(2) a set of character glyphs; 

15 (b) processing logic, coupled to the recognition 

logic, for output ing to the display a non- 
Roman character corresponding to a character 
glyph and from zero to two pre-character 
modifier glyphs* 

20 21. An improved electronic handwriting 

recognition system for interpreting input strokes and 
displaying corresponding katakana characters, the system 
being of the type having an input writing surface, a 
device for inputing strokes on the input writing surface, 
25 and a display for displaying characters, the improvement 
comprising: 

(a) recognition logic for recognizing and 

classifying each individual input stroke 
after input as a member of one of a plurality 
30 of sets of glyphs, the sets comprising at 

least: 

(1) a set of pre-character modifier glyphs; 

(2) a set of character glyphs; 
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(b) processing logic, coupled to the recognition 
logic, for outputing to the display a 
katakana character corresponding to a 
character glyph and from zero to two pre- 
5 character modifier glyphs. 



22. An improved electronic handwriting 
recognition system for interpreting input strokes and 
displaying corresponding katakana characters, the system 
being of the type having an input writing surface, a 
10 device for input ing strokes on the input writing surface, 
and a display for displaying characters, the improvement 
comprising: 

(a) recognition logic for recognizing and 
classifying each individual input stroke 

15 after input as a member of one of a plurality 

of sets of glyphs, the sets comprising at 
least: 

(1) a set of pre-character modifier glyphs 
corresponding to Roman character 

2 0 consonants; 

(2) a set of character glyphs corresponding 
to Roman character vowels; 

(b) processing logic, coupled to the recognition 
logic, for outputing to the display a 

25 katakana character corresponding to a 

character glyph and from zero to two pre- 
character modifier glyphs. 
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